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ABSTRACT 

Noting that program evaluators can profit by adopting 
the investigative journalist's goal of discovering hidden 
information, this guide explores the journalist's investigative 
process — without its element of suspicion — and discusses hov 
components of this process can be applied to program evaluation. 
After listing the major characteristics of the investigative 
approach, the guide describes the use of an investigative posture in 
evaluation, lists criteria for determining when to use investigative 
journalism methods, outlines steps for carrying out an investigative 
program evaluation, and provides an example of investigative 
journalism techniques applicable for evaluation, using the Keller 
Plan teaching method (observation, interviews, analysis of records, 
and an interview to determine responses to previous data). Finally, 
the guide describes ethical and legal concerns of investigative 
journalism applicable to program evaluators and concludes with a 
glossary of terms commonly used in the investigative journalism 
field. (HTH) 
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The job of the investigative reporter is to 
ex<>mine all the institutions of society and 
report how they work«*-not how they were 
designed to work or how their leaders claim 
they work, but how they re^^lly work. 

Willidi^s, 1978r p. 188 



CHARACTEiaSTICS OF THE INVESTIGATIVE APPROACH 



The investigative reporter is the canker in 
the sore of the official mouth . . . 

Urban Policy Research Institute 



The stories of two little-known reporters of the Washington 
Post, Carl Bernstein and Bob Woodward, precipitated a series of 
historical events that instantly endowed the investigative 
reporter with celebrity status. Digging beneath the surface of 
of government life, they discovered a corrupt world of immense 
proportions. Their stories resulted in the indictment of high 
White House officials, led to the resignation of the President, 
and raised f undeunental doubts about the institutions and laws 
that had allowed Buch corruption to grow unchecked. 

Woodward and Bernstein are part of an American tradition of 
authors who have made it their purpose to question what they 
see. In writing about everything from hot dogs, to funerals, to 
Corvairs, investigative journalists share a common commitment to 
doubt and suspicion. The investigative reporter operates under 
what Douglas (1976) describes as 

... the assunqption that profound conflicts of 
interests, values, and feelings pervade social life. 
It is taken for granted that many of the people one 
deals with, perhaps all people to some extent, have 
good reason to hide from others what they are doing 
and even to lie to them. Instead of trusting people 
and expecting trust in return, one suspects others and 
expects others to suspect him. Conflict is the 
reality of life; suspicion is the guiding principle. 

(p. 55) 

The major characteristics of the investigative approach 
include the following: 

• The goal of the investigative journalist is to discover 
and then uncover information which is important and 
hidden. 



The investigative journalidt assumes defensiveness and 
non-cooperation. 



• The investigative journalist jnaxiraizes the use of 
existing records as well as his/her own observations and 
interviews. 

• As the investigation proceeds, the reporter continuously 
checks that, the story remains feasible, significant, and 
acceptable. 

• Great attention is paid to thoroughness, soundness , 
depth, and originality of research. 

• In writing the final story, the reporter strives for 
fairness, not objectivity. 

• The investigative reporter hs« a systemic focus and 
attempts to illustrate deeper issues with the specific 
case he/she is studying. 



Usually the evaluator does not need to operate with the same 
decree of suspiciousness, and yet the evaluator nay at times 
profit by adopting the journalist's goal of discovering 
information that is hidden. The journalist is looking for 
information that is hidden with malicious intent; the evaluator 
looks for realities that are hidden by surface descriptions, by 
the need of programs to present their best face, or by simple 
lack of awareness. 



There are three ways in which evaluators might profit from 
studying the investigative reporter. Evaluators can 

1. assume the reporter's investigative posture; 

2> adapt some of the reporter's methods and tools; 

3. learn from the reporter's experience with legal and 
ethical questions. 



The Investigative Posture in Evaluation 

Many times an evaluator begins the study of a program by 
siaply observing it. After a while a feeling may emerge— a 
perception that something is happening (or not happening) that is 
inconsistent with the stated purposes of the program. There 
appears to be a discrepancy between the design and the 
actual i^~between the espoused philosophy and the existing 
program practices, or between the described program operations 
and the actual ones. 



Adopting the posture of the investigative reporter ^ the 
evaluatot can begin to substantiate his/her perceptions by 
observations r interviews ^ and record searches* The evaluator 
needs to document that what he/she sees is (1) a consistent 
pattern (not an anomaly), (2) a significant pattern, and (3) an 
aspect of the program that is not generally known or 
acknowledged. As a final step, the evaluator can present his/her 
findings to the program director and other key participants^ and 
probe their reactions in an in-depth interview* Out of this 
investigative process the evaluator will hopefully gain some 
insight into what is happening and will generate a deeper and 
more honest understanding of the program. 



KNOWING WKSN TO USE THE METHODS OF 
INVESTIGATIVE JOURNALISM 

Provided below is a list to assist the evaluator in 
determining whether or not to use the methods of investigative 
journalism in an evaluation. Use these methods when 

• the nature of the evaluation calls for a non-intrusive study 
of ongoing processes as they occur in their nattural setting; 

• thence are individuals or organizations who are suspected of 
being incompetent (or being guilty of wrongdoing) and who may 
be defensive or uncooperative; 

• the functioning of a program or an institution appears to be 
quite different from what it is presented to be or supposed 
to be; 

• key information is buried in existing records; 

• it is suspected that more systemic or fundcunental problems 
underlie observed events and issues; 

• an evaluation turns up unexplained major issues that go 
beyond the original evaluation mandate. 



STEPS IN CARRYING OUT AN INVESTIGATIVE EVALUATION 



This section outlines the major steps a journalist uses in 
carrying out an investigation. Most of them are techniques the 
evaluator may use or adapt for a particular evaluation (see 
Figure 1 and the glossary for more details). 
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1. Conception 



Germinal ideas can come from outside sources ( tipsters # 
colleagues, and other leads), inside sources (leads from 
other work; spin-offs from other investigations), and 
blue-sky sources (serendipity and the acute "smell" of the 
intuitive reporter). The original conception of a story may 
be well defined or may be just a sense ("there is something 
rotten in the state of Denmark"). 



2. Feasibility Check 

Reporters may initially prepare a memo for editors with 
answers outlined to the following questions: 

Is it possible to do the story? 
Do we have the required personnel and technical 
expertised 

Do we have the funds and other resources? 

Do we have time? Will the story sit still long enough? 

What are the barriers to getting the full story? 

Wiiat are the barriers, to publishing it? 

What is the ultimate significance of the story? 

3. Min/Max Pro1ections~<So/No-Go Decision 

The reporter estimates minimum and ^xiroum story 
possibilities. Decisions are made with the editor about 
whether to proceed, postpone, or abort. 

4. Planninq/Base-Building 

Personnel are assigned; files are set up; methods are 
outlined and deadlines determined. An initial attempt is 
made to formulate boundaries and a central focus for the 
study. 

5. Original Research 

Reporters engage in iterative cycles of hypotheses 
formulation, tracking (record searches), interviews, and 
observation. Profiles are developed and findings summarized. 

6. Re-Evaluation — Go/No-Go Decision 

Pending the outcc»ie of the original research^ the 
investigation may be given a green light, postponed, 
redirected, or aborted. 
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7; Kgy interviews 



Armed with the information gained so far, the investigators 
go to the focus (targets) of the story, seeking confirmation, 
denial, confrontation, and breakthroughs. 

8. Re-Evaluation— Go/NO"Go Decision 

Should the story now be written up and published? 

9. Reporting 

The reporter chooses a central theme and writes a soundly 
documented story around it; he/she makes final 
substantiations and thoroughly checks all proofs. 



Tnvestigative reporting is the uncovering of 
facts that someone — I would say substantial 
facts — that someone or some organization wants 
to keep hidden. That is part one of the 
definition. Part two of the definition is that 
it roust be your own work. 

Bob Green, Newsday 



AN EXAMPLE: AN EVALUATION OF A 
KELLER PLAN COURSE 



An example of using investigative journalism techniques for 
evaluation is described below. It concerns the evaluation of the 
Keller Plan teaching method. 



The Keller Plan is a course format in which students learn 
step-by-step as they progress at their own pace. Course material 
is divided into digestible learning modules, and students must 
deaonstrate mastery of each module (by passing a test) before 
proceeding. In the evaluation of one such course, an 
investigative approach used observations, interviews, and careful 
examination of course records to see how closely the course 
actually followed the Keller Plan model. 



Observation 

After several weeks of watching the course in operation, the 
following variations in .the Keller Plan aodel were observed: 

• Tutors (who helped students and graded the K>dule tests) 

foraed close working (personal) relationships with students. 
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• The teaching and grading functions were inter-mixed. (The 
"grading" of the short module tests was taking an average of 
30 minutes.) 

• The grading of a test was as much a matter of judgment on the 
part of the tutor as it was an algorithmic procedure. 

Interview s 

Interviews with tutors and students yieldad the following 
additional insights: 

• Tutors felt social pressure to pass the students: "if you 
pass him you feel like you shouldn't have; if you fail him, 
you feel like a bad guy. It's a no-win situation^." 

• Students felt pressure to "keep on schedule" — a pressure 
which they passed on to the tutors: "I would talk to the 
student first and find out what schedule he was on. If he 
was way behind, I would pass him; otherwise I would ask him 
to try again." 

• Tutors felt a pressure from the instructor to be more 
strict: "When the instructor is around the course center, 
you feel conflict because you know your way isn't the way he 
would do it. You know you are supposed to be doing it from 
his point of view. " 

• Tutors felt their teaching function was more important than 
grading: "Grading is subordinate to teaching. I do most of 
my teaching in grading sessions, because there is so much to 
do, and it is the only time I really get to see the people. 
Grading is a very important part of the learning process." 

Analysis of Records 

A review of the course test records revealed several other 
interesting and related facts: 

• Many students failed the first test on a module and then 
immediately passed a second. (It turns out they used the 
first test as a practice and as a study guide.) 

• Before the mid-term and end-of-course deadlines, the nund>er 
of tests taken greatly increased, the time between tests 
decreased, and the ratio of first-time passes greatly 
increased. (Apparently tutors and students worked together 
to get everyone through the course on time.) 



Key Interview 



The information gained from observations, interviews, and 
record analyses presented a picture of the course that varied 
considerably from the self -paced, step-by-step mastery described 
by Keller. When the instructor was presented with the findings 
of the study, his reaction was: 

At the present time I am living with this phenomenon, 
and I would say I am not pleased. You know, I am not 
satisfied with the fact that this is happening, but I 
am not sure whether anything should be done about it, 
and to what extent something could be done about it • 
. . I have to judge to what extent imposing external 
constraints (which is all I could do) would work. 

While the investigation did not lead to immediate solutions 
of the problems uncovered, it did dispel an illusion — the ideal 
of the Keller Plan model — so that what was actually happening 
became clearer. 



ETHICAL AND LEGAL CONCERNS 



This section describes the concerns that the investigative 
reporter must deal with: ethical, legal, and systemic* Many are 
concerns for the evaluator as well. 



Ethical Concerns 

From time to time, the investigative reporter may engage in 
one or more of the following practices in order to break the 
protective surface that hides the information he/she seeks: 

• covert taping of phone calls or interviews; 

• concealing true intentions, motives, or identities; 
4) accepting gifts; 

• riding piggyback on another's interest; 

• paying for infornation; 

• changing faces and postures to bi^ agreeable to different 
interests; 

• imposing covert surveillance; 

• invading privacy; 
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• breaching confidences; 



• quoting out of context and/or selecting information to 
advance one's story. 



Although these practices could be considered unethical, 
depending on the observer and the context of their use, the 
issues they raise are clearly relevant to the evaluator. 
Unfortunately, journalists' experience sheds only a little light 
on standards for the use of these practices. The following very 
general and conditional guidelines have emerged: 

1. Avoid deceptive behavior where possible. 

2. Be sure there are no other alternatives before engaging in 
deceptive behavior. 

3. Err on the side of honesty* 

4. Determine standards (what is permissible/not permissible) 
ahead of time. 

5. Make sure the gain in the public interest that results from 
using deceptive practices outweighs the cost of using these 
practices. 



Legal Concerns 

The investigative reporter has encountered legal battles over 
access and protection of information. Freedom of information 
laws have provided access to many files and records previously 
classified as private. On the other hand, they may have made the 
files of the evaluator a matter of public record. Consequently, 
the protection of sources and guarantee of confidentiality may be 
something the evaluator can no longer give with total assurance. 
Finallyr the evaluator aay be able to learn from the libel law 
experience of the journalist — evaluators may need to take care to 
avoid suits for defamation or even for falsely "accrediting" a 
program lAich later proves to be lacking. 



Systoaic Concerns 

Finally, journailists* systemic concern may be of example to 
the evaluator* Throughout the investigation and the writing of 
the story, the journalist is seeking to ground his/her story in 
deeper issues. Ultimately, the journalist seeks significant and 
wide^read reform. Consequently, in addition to carrying out 
research and reporting the story, journalists make contact and 
personal connecticMis with influential peqple who can do something 




about the story they are writing. The journalist often suggests, 
by implication if not directly, actions that would lead to reform 
and long-term solutions. 



CAVEATS 



An evaluator using investigative journalism techniques may 
encounter difficulties, including the following: 

• Adopting an investigative posture may lead to increased 
defensiveness and conflict with those being evaluated* 

• Ik>th the methods and products of an investigative approach 
may be considered unacceptable as evaluation* 

• The investigation may well lead beyond the bounds of the 
originally stated evaluation mandate « 

• Probing for deeper issues may uncover fundamental 
"unsolvable" difficulties — findings not welcomed by any of 
the evaluation audiences. 

• Considerable resources may be expended on discovering a story 
that appears relevant and significant only to the evaluator. 

• Adopting a guideline of fairness instead of objectivity may 
not be comprehensible or acceptable to the evaluation sponsor 
and audiences. 

• Ethical questions of method may arise. 

• The evaluator may be led astray by individuals with "axes to 
grind,* or may be led to overstate the importance of the 
problems he/she discovers. 

• There are usually consequences of revealing what someone may 
have wanted hidden. 



A GLOSSARY OF TERMS 



Like all professional fields, investigative journalism has 
developed its own key words and phrases. This glossary gives a 
sample of some of the most common terms, along with examples of 
useful evaluation techniques r> 

The Fast Study 

This is the iiuitial I^se of an investigation where the 
reporter seeks to learu the territory. Through a review 
of what has been written# a reporter can avoid repeating 
irtiat has alrcuMl^ been done atnd can find pronisix^; points 
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of entry. Through a brief immersion in the scene, a 
reporter seeks to learn the laws and norms of the 
setting and to discover the perspective? of those 
involved. Also, the reporter may discover any hidden 
motivations or constraints behind the commissioning of 
the story. A brief fast study will aid greatly in 
formulating a reasonable investigation and may avoid 
much wasted effort. 

Minimum/Maximum Goal Projection 

After a fast study, and before beginning original 
research, reporters may formulate minimum and maximum 
projections for the story. The maximum is what they 
might get if all their hunches are borne out; the 
minimum is what they are almost assured of learning. 
(Before beginning an evaluation, evaluators might 
consider discussing similar projections with their 
clients.) 

Evaluation examples of min/max projections: 

1. At a minimum, we will be able to measure how well 
Students mastered the basic math skills the progr:;iQ 
has specified it would teach. At a maximum, we will 
be able to certify that students gained some more 
general problem solving skills through their work in 
this program. 

2. At a minimum, we will provide a 90od (thick) 
description of the program in action. At a maximum, 
we will be able to report insights that yield 
creative suggestions for future actions. 

Tracking 

Tracking is the main technique used in the initial 
research phase of the investigation. Starting with a 
hunch (hypothesis) in mind, the investigator looks for 
evidence that confirms or denies his/her suspicions. 
The reporter must use all of his/her worldly knowledge 
and creativity in imagining where and how tracks would 
be left behind. The ingenious mind may find many 
recorded evidences that tend tc corroborate or deny 
proposed hypotheses. (Evaluators tend to always design 
their own insttuments; they often underutilize 
documents, records, and ether existing sources of 
information. ) 

Evaluation example: 

An undergraduate physics course at a large university 
was to be taught entirely through a CAI (computer 
assisted instruction) method. In addition to the 
main programmed sections, there were optional 
sophisticated cooiputer tutorials which taught a 
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variety of related physics topics. Developed under a 
National Science Foundation grant, these tutorials r 
and their use in an undergraduate course, were to be 
evaluated by an independent evaluator. 

The first look at computer-use records showed that 
the tutorials were getting many hours of student 
use. However, this did not agree with the 
evaluator 's assessment that the students' knowledge 
of the tutorial content was poor. Through a more 
careful look at the use records, it was seen that 
most students tried o\:\ (entered) most of the 
tutorials, but 9^ percent of the tutorials were 
abandoned in less than 6 minutes. Only 5 percent of 
the tutorial interactions lasted for more than 30 
minutes, which was judged to be a minimal time for 
significant interaction to have occurred. Almost all 
students were browsing and putting in an appearance. 

Thus, the careful review of existing records — in this 
case the student<-use pattern of the computer 
tutorials — gave a less optimistic picture than the 
original report suggested. 

Modus Operandi 

Closely related to tracking, modus operandi refers 
literally to a method of operating. (The mass murderer 
is identified with the specific trademarks of his 
kills.) Even when proof is not available, an 
accumulation of identifying clues, or a chain of 
connecting evidence, may be enough to substantiate a 
hypothesis or to help the reporter decide between 
alternatives. 

Evalaition example: 

In a student problem-solving course, the experimental 
group did not do significantly better than a control 
group on a difficult set of problems. However, a 
close examination of the scratch-mork of the two 
groups revealed that the students in the problem- 
solving class had learned to use several heuristic 
diagrams in flexible ways. 

Boundaries and Filling 

Early in the investigation a boundary is determined so 
that investigative efforts can be limited to relevant 
issues. Once boundaries concerning people, dates, 
issues, and events have been agreed upon, it becomes 
clearer where there are missing links. Then interviews, 
observations, and record searches can be directed to 
"fill" the gaps. 
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Circling 

In an interview, new information is obtained from one 
source. The reporter then circles this information 
around his/her other sources seeking confirmation and 
additions. Circling provides a natural way in an 
interview to introduce topics that may be uncomfortable. 

Shuffling 

This process is similar to circling, but more effort is 
made to build upon the information being shuffled 
around. Thus, material is presented to A; the 
interaction with A is recorded and the information 
modified as indicated. The altered version (or both 
versions} are presented to B for reaction; B*s denial or 
confirmation is presented back to A or is taken on to C, 
and so on. ... 

Triangulation 

The reporter looks for convergence of information 
gathered from different sources. If a proposition can 
survive the onslaught of a series of imperfect measures 
and checks, then it probably merits credibility. 

Filing 

Files are an integral tool of the investigative 
journalist. A separate file for each key individual is 
maintained and all files are cross-referenced. As 
information is collected, summaries are written and 
"profiles* developed. Complete, accurate, and organized 
files are important in helping the reporter to see 
possible connections, to draw accurate conclusions, and 
to substantiate inferences. 

Key Interview 

The culmination of the investigative process is the key 
interview. Those individuals central to the story, "the 
targets," are interviewed in order to confirm or deny 
the information that has already been put together. 
And, more importantly r the key interview is used to coax 
out additional critical information. 

The six stages of the key interview have valuable 
analogues for the evaluator: 

1. Gaining entry : Most individuals, even the most 
uncooperative, are reluctant to admit that they are 
unwilling to talk to the press, to present their own 
story. 

2. Preparin g: Rereading files, formatting the 
Interviewr preparing and ordering questions, and role 
playing are all preparatory means to make the most of 
the interview. 



3c Opening the interview ; The interview is typically 
begun in a easy, non* threatening manner. The ground to 
be covered should be outlined, and rules of fair play 
indicated, if needed. 

4. Controlling the interview ; Various techniques are 
used to keep control of the interview and to avoid 
sidetracks~feigning ignorance or naivete (the Colombo 
approach), understating the case^ overstating the case 
(and thus eliciting confession to a less serious 
offense) , hinting at greater knowledge than one actually 
possesses, and shock (direct and abrupt confrontation 
with facts the target is likely to think are hidden). 

5. Confronting the target s The investigator maintains 
courtesy while increasing the pressure. Presenting 
documentation, becoming more specific, demonstrating 
contradictions, and even registering indignation are all 
ways of bringing the target to the heart of the matter. 

6. Recording s As a backup, the interview should be 
(openly) recorded. 

Whenever possible, the key interview should be done by a 
pair of reporters to increase pressure and accuracy of 
perceptions. 

Examples The following are a reporter's recollection of 
a critical moment in a key interview. 

... at about the middle of the interview, you 
talk about intimidation and counterintimida*- 
tion, he had been bottled up so long, wanting 
in his egotistical way to show us how wrong we 
were . . . that he starts answering questions 
way beyond what he should ... We like to 
feel that when you go in to ask the questions 
you know 90 percent of the ansi^c^rs. Just the 
last 10 percent is what you are going to get if 
you are lucky. If not, you at least get self- 
serving statements that give you balance for 
your story. . . But he starts spilling stuff 
we hadn't heard— stuff that would put him in 
jail. 

Finally, his attorney passes him a note — 
•for Christ's sake — shut upi* 

Urban Policy Reseatch Institute, 
1976 r pp. 13ff. 

Reporting 

In writing their story f investigators must keep the follou^ing 
questions in the forefront of their ninds. 
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• What is this story ^bout? What am I trying to prove? 

• Who gives a damn? 

• Why will they care? 

(How many evaluations would be vastly improved by 
adherence to these questions?) 



Follow-Through 

Reporters are careful to monitor the production of their 
story, right up through the proofreading of the galleys, 
guarding carefully that no inaccuracy or error sneaks 
in. Reporters also stimulate public reaction to the 
story and maximize its impact where possible, so that it 
does more thrn create a short-term sensation. 

Aborting 

At several checkpoints during the investigation, the 
reporter makes a go/no-go decision. Many stories are 
aborted because they become unfeasible, too difficult, 
ill-advised, or of insufficient interest or significance. 
(Evaluators might do well to consider the abortion, or 
at least redirection, option as they proceed). 

Teaming 

Many reporters work in pairs, which has the advantage of 
checking emerging interpretations so that false 
projections are more quickly recognized and dropped. 
Other advantages of the team approach include the 
sharing of expertise, energy^ and perceptions. 



Fairness 

Fairness replaces objectivity as the standard by which 
the validity of the investigation and story is to be 
judged. A fair stor^ is one that presents all points of 
view from the advocate's position fully and without 
distortion. The ultimate "truth* of a story is of less 
concern to the reporter than telling a story which is 
substantiated and fairly represents all sides. 
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